Distribute LLVM bitcode linker as a preview component #123423

kjetilkjeka · 2024-04-03T16:54:55Z

The self-contained LLVM bitcode linker was recently added in #117458. It is currently only in use to link the Nvidia ptx assembly tests when running rustc tests (local or CI). In fact, the linker itself is currently only usable for the nvptx64-nvidia-cuda target, but more targets will be supported in the future.

The reason a new linker was needed for the ptx format is that the old one has not been updated the last few years. It worked fine for a while, but as LLVM changed it broke and the nvptx tests was disabled in rustc back in 2019. It was ad-hoc patched and have been used in a sub-optimal state by the community until now.

If this PR is merged, the LLVM bitcode linker will be distributed as a preview component that can be used as a replacement for the old ptx-linker for development in addition to rustc tests. In addition to installing the llvm-bitcode-linker component, also the llvm-tools component must be installed as the llvm-bitcode-linker works by calling llvm tools.

Even though the LLVM bitcode linker is in its early stages it already now provides a lot of value over the old ptx-linker just by working and using up-to-date llvm tooling. By shipping it as a component it will be easier to gather user experience and improving it.

@petrochenkov when installing as a component it will be installed in the self-contained folder and will not work with -Clink-self-contained=no (although for some reason I expect to be a bug -Clink-self-contained=-linker doesn't properly disable it). However, when building using x.py build it will be placed in the lib/rustlib/<target>/bin directory and will be available for internal tests even if -Clink-self-contained=no is passed.

CC: @Mark-Simulacrum as I very briefly discussed it with you some months ago https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/.E2.9C.94.20How.20to.20ship.20a.20new.20tool.20.28embedded.20linker.29.20to.20users.3F

rustbot · 2024-04-03T16:55:03Z

r? @clubby789

rustbot has assigned @clubby789.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

src/bootstrap/src/core/build_steps/install.rs

clubby789 · 2024-04-06T00:08:32Z

lgtm other than the typo. Since Mark originally suggested it I'm going to assign them to make sure this is the right procedure
r? @Mark-Simulacrum

Mark-Simulacrum

r=me with commits squashed

Mark-Simulacrum · 2024-04-13T13:41:39Z

src/bootstrap/src/utils/tarball.rs

@@ -64,6 +65,9 @@ impl OverlayKind {
                "compiler/rustc_codegen_cranelift/LICENSE-APACHE",
                "compiler/rustc_codegen_cranelift/LICENSE-MIT",
            ],
+            OverlayKind::LlvmBitcodeLinker => {


Why is this a separate/new overlay kind rather than just using OverlayKind::Rust?

The idea was to provide the readme of the llvm-bitcode-linker. I have corrected the path to point to this readme file now.

rustbot · 2024-04-15T12:59:18Z

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred in match lowering

cc @Nadrieril

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Mark-Simulacrum · 2024-04-15T13:41:54Z

@bors r+ rollup=iffy

bors · 2024-04-15T13:41:57Z

📌 Commit 235d45e has been approved by Mark-Simulacrum

It is now in the queue for this repository.

…mponent, r=Mark-Simulacrum Distribute LLVM bitcode linker as a preview component The self-contained LLVM bitcode linker was recently added in rust-lang#117458. It is currently only in use to link the Nvidia ptx assembly tests when running rustc tests (local or CI). In fact, the linker itself is currently only usable for the `nvptx64-nvidia-cuda` target, but more targets will be supported in the future. The reason a new linker was needed for the ptx format is that the [old one](https://github.com/denzp/rust-ptx-linker) has not been updated the last few years. It worked fine for a while, but as LLVM changed it broke and the nvptx tests was [disabled in rustc back in 2019](rust-lang@f8f9a28). It was ad-hoc patched and have been used in a sub-optimal state by the community until now. If this PR is merged, the LLVM bitcode linker will be distributed as a preview component that can be used as a replacement for the old ptx-linker for development in addition to rustc tests. In addition to installing the `llvm-bitcode-linker` component, also the `llvm-tools` component must be installed as the `llvm-bitcode-linker` works by calling llvm tools. Even though the LLVM bitcode linker is in its early stages it already now provides a lot of value over the old ptx-linker just by working and using up-to-date llvm tooling. By shipping it as a component it will be easier to gather user experience and improving it. `@petrochenkov` when installing as a component it will be installed in the self-contained folder and will not work with `-Clink-self-contained=no` (although for some reason I expect to be a bug `-Clink-self-contained=-linker` doesn't properly disable it). However, when building using `x.py build` it will be placed in the `lib/rustlib/<target>/bin` directory and will be available for internal tests even if `-Clink-self-contained=no` is passed. CC: `@Mark-Simulacrum` as I very briefly discussed it with you some months ago https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/.E2.9C.94.20How.20to.20ship.20a.20new.20tool.20.28embedded.20linker.29.20to.20users.3F

Rollup of 12 pull requests Successful merges: - rust-lang#123423 (Distribute LLVM bitcode linker as a preview component) - rust-lang#123548 (libtest: also measure time in Miri) - rust-lang#123666 (Fix some typos in doc) - rust-lang#123864 (Remove a HACK by instead inferring opaque types during expected/formal type checking) - rust-lang#123896 (Migrate some diagnostics in `rustc_resolve` to session diagnostic) - rust-lang#123919 (builtin-derive: tag → discriminant) - rust-lang#123922 (Remove magic constants when using `base_n`.) - rust-lang#123931 (Don't leak unnameable types in `-> _` recover) - rust-lang#123933 (move the LargeAssignments lint logic into its own file) - rust-lang#123934 (`rustc_data_structures::graph` mini refactor) - rust-lang#123941 (Fix UB in LLVM FFI when passing zero or >1 bundle) - rust-lang#123957 (disable create_dir_all_bare test on all(miri, windows)) r? `@ghost` `@rustbot` modify labels: rollup

bors · 2024-04-15T19:14:19Z

⌛ Testing commit 235d45e with merge 99d0186...

Rollup merge of rust-lang#123423 - kjetilkjeka:llvm_bitcode_linker_component, r=Mark-Simulacrum Distribute LLVM bitcode linker as a preview component The self-contained LLVM bitcode linker was recently added in rust-lang#117458. It is currently only in use to link the Nvidia ptx assembly tests when running rustc tests (local or CI). In fact, the linker itself is currently only usable for the `nvptx64-nvidia-cuda` target, but more targets will be supported in the future. The reason a new linker was needed for the ptx format is that the [old one](https://github.com/denzp/rust-ptx-linker) has not been updated the last few years. It worked fine for a while, but as LLVM changed it broke and the nvptx tests was [disabled in rustc back in 2019](rust-lang@f8f9a28). It was ad-hoc patched and have been used in a sub-optimal state by the community until now. If this PR is merged, the LLVM bitcode linker will be distributed as a preview component that can be used as a replacement for the old ptx-linker for development in addition to rustc tests. In addition to installing the `llvm-bitcode-linker` component, also the `llvm-tools` component must be installed as the `llvm-bitcode-linker` works by calling llvm tools. Even though the LLVM bitcode linker is in its early stages it already now provides a lot of value over the old ptx-linker just by working and using up-to-date llvm tooling. By shipping it as a component it will be easier to gather user experience and improving it. ``@petrochenkov`` when installing as a component it will be installed in the self-contained folder and will not work with `-Clink-self-contained=no` (although for some reason I expect to be a bug `-Clink-self-contained=-linker` doesn't properly disable it). However, when building using `x.py build` it will be placed in the `lib/rustlib/<target>/bin` directory and will be available for internal tests even if `-Clink-self-contained=no` is passed. CC: ``@Mark-Simulacrum`` as I very briefly discussed it with you some months ago https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/.E2.9C.94.20How.20to.20ship.20a.20new.20tool.20.28embedded.20linker.29.20to.20users.3F

kjetilkjeka · 2024-04-16T08:40:45Z

When checking if this component made it into nightly today I'm running

$ rustup +nightly-2024-04-16 component list | grep llvm
llvm-tools-x86_64-unknown-linux-gnu

And the llvm-bitcode-linker is not picked up. I thought rustup would pick up on new available components. Is that wrong and something must be done in rustup as well?

Edit: I guess it's not rustup as I cannot find the component here as well https://static.rust-lang.org/dist/2024-04-16. Then I assume it must be something related to deployment scripts.

Edit2: It seems to me like the distribution script is running python3 x.py dist bootstrap --include-default-paths and copying files. This should include the llvm-bitcode-linker so I'm a bit confused to why it doesn't happen.

…ild_manifest, r=Mark-Simulacrum Add llvm-bitcode-linker to build manifest When creating rust-lang#123423 I didn't realize I also had to add the new component to the build-manifest. This PR finishes the work of adding it, by also adding it to the build manifest. r? `@Mark-Simulacrum`

…ild_manifest, r=Mark-Simulacrum Add llvm-bitcode-linker to build manifest When creating rust-lang#123423 I didn't realize I also had to add the new component to the build-manifest. This PR finishes the work of adding it, by also adding it to the build manifest. r? ``@Mark-Simulacrum``

Rollup merge of rust-lang#124071 - kjetilkjeka:llvm_bitcode_linker_build_manifest, r=Mark-Simulacrum Add llvm-bitcode-linker to build manifest When creating rust-lang#123423 I didn't realize I also had to add the new component to the build-manifest. This PR finishes the work of adding it, by also adding it to the build manifest. r? ``@Mark-Simulacrum``

rustbot assigned clubby789 Apr 3, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Apr 3, 2024

This comment has been minimized.

Sign in to view

kjetilkjeka force-pushed the llvm_bitcode_linker_component branch from 9922d1e to a4ef243 Compare April 4, 2024 09:28

clubby789 reviewed Apr 6, 2024

View reviewed changes

src/bootstrap/src/core/build_steps/install.rs Outdated Show resolved Hide resolved

rustbot assigned Mark-Simulacrum and unassigned clubby789 Apr 6, 2024

Mark-Simulacrum reviewed Apr 13, 2024

View reviewed changes

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 13, 2024

kjetilkjeka force-pushed the llvm_bitcode_linker_component branch from 24e140b to 30e6af0 Compare April 15, 2024 12:59

rustbot added the O-wasi Operating system: Wasi, Webassembly System Interface label Apr 15, 2024

Distribute LLVM bitcode linker as a preview component

235d45e

kjetilkjeka force-pushed the llvm_bitcode_linker_component branch from 30e6af0 to 235d45e Compare April 15, 2024 13:39

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 15, 2024

This was referenced Apr 15, 2024

Rollup of 13 pull requests #123966

Closed

Rollup of 12 pull requests #123968

Merged

bors merged commit e07d18f into rust-lang:master Apr 15, 2024

rustbot added this to the 1.79.0 milestone Apr 15, 2024

kjetilkjeka mentioned this pull request Apr 17, 2024

Add llvm-bitcode-linker to build manifest #124071

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Distribute LLVM bitcode linker as a preview component #123423

Distribute LLVM bitcode linker as a preview component #123423

Uh oh!

kjetilkjeka commented Apr 3, 2024 •

edited

Loading

Uh oh!

rustbot commented Apr 3, 2024

Uh oh!

This comment has been minimized.

Uh oh!

clubby789 commented Apr 6, 2024

Uh oh!

Mark-Simulacrum left a comment

Uh oh!

Mark-Simulacrum Apr 13, 2024

Uh oh!

kjetilkjeka Apr 15, 2024

Uh oh!

rustbot commented Apr 15, 2024

Uh oh!

Mark-Simulacrum commented Apr 15, 2024

Uh oh!

bors commented Apr 15, 2024

Uh oh!

bors commented Apr 15, 2024

Uh oh!

kjetilkjeka commented Apr 16, 2024 •

edited

Loading

Uh oh!

Uh oh!

Distribute LLVM bitcode linker as a preview component #123423

Distribute LLVM bitcode linker as a preview component #123423

Uh oh!

Conversation

kjetilkjeka commented Apr 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Apr 3, 2024

Uh oh!

This comment has been minimized.

Uh oh!

clubby789 commented Apr 6, 2024

Uh oh!

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Uh oh!

Mark-Simulacrum Apr 13, 2024

Choose a reason for hiding this comment

Uh oh!

kjetilkjeka Apr 15, 2024

Choose a reason for hiding this comment

Uh oh!

rustbot commented Apr 15, 2024

Uh oh!

Mark-Simulacrum commented Apr 15, 2024

Uh oh!

bors commented Apr 15, 2024

Uh oh!

bors commented Apr 15, 2024

Uh oh!

kjetilkjeka commented Apr 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kjetilkjeka commented Apr 3, 2024 •

edited

Loading

kjetilkjeka commented Apr 16, 2024 •

edited

Loading